Introduction
The Fleek Network node streams text messages to the standard output and error in Linux. Given that disk space is a limited resource for most systems, the number of text data can be a cause of concern. Managing the file sizes by means of rotation and compression can help.
A conventional Fleek Network Service setup (as per the documentation and tool recommendations), have the text data stored in a special directory for storing logs called /var/log. The /var/log directory contains logs from various applications running on the operating system, the operating system itself, and others.
For the purpose of storing network node operation logs, it defaults to the /var/log/lightning directory where the files output.log (stdout) and diagnostic.log (stderr) is located and accumulated. As the node runs, the size of the files increase, as it aggregates all the output generated by the Fleek Network service operations, such as info, errors, etc.
A Node Operator can configure the system to rotate, compress, and set maximum size of these files to safeguard any concerns or issues that can get out of control, such as causing disk space to become full quickly.
In this guide, we'll look into some options available to manage the logs. First, we'll look into journald.conf which controls where to store journal data (the journal is a component of systemd that handles all the messages in a Systemd enabled system). Afterwards, we'll look into Logrotate an application to help us manage automatic rotation and compression of log files.
In essence, the journal and the logs duplicate the same information, and we want to make sure that we set measures to control it.
Journal
Journal is a component of Systemd, a centralized location for all messages logged by different components in a systemd-enabled Linux system. The systemd journal will happily run in parallel with the standard type log files in /var/log/* where the Fleek Network Systemd Unit Service outputs Standard Output and Standard Error in the location /var/log/lightning/*.log.
Here, we are going to learn how to configure the journald.conf service configuration file for the system-wide settings (meaning that it applies to all services), but firstly we're going to learn some commands to help us troubleshoot when necessary.
Commands
Disk utilization checkup
To check how much disk space is used by Systemd log files, run the command below:
sudo journalctl --disk-usage
It provides information of how much disk space is utilized by the log files in your system.
Clearing logs manually
The best way to clear log files is done automatically by the journald.conf configuration file, discussed in Configuration file. In the ideal world, you shouldn't have to manually delete the log files, but this can be useful to know about when troubleshooting.
To flush the log files run the command below:
sudo journalctl --flush --rotate
sudo journalctl --vacuum-time=1s
The flush and rotate flag is used, as vaccum-time only clears archived logs and not active ones. It'll flush:
- Move /run/log/journal to /var/log/journal
- Rotate (this flag archives logs and retains it)
Since it'll only keep the past 1-second-long files, it'll effectively clear everything.
Follow or tail logs of service
sudo journalctl -u <SERVICE-NAME>.service -f
For example, for a conventional native install (if you haven't followed the conventions, make the appropriate tweaks to fit your needs).
sudo journalctl -u lightning.service -f
The Docker service
sudo journalctl -u docker-lightning.service -f
Show all service entries
Show all journal entries, which can be fairly long:
sudo journalctl
Configuration file
The default configuration file is located at /etc/systemd/journald.conf. This is the main file that journal reads the configuration from, for the system instances that are controlled by root.
In addition to the main configuration file there are a few other locations that take higher precedence and override the main configuration file. To learn more about journald read here.
To keep the guide short we are going to use the main location /etc/systemd/journald.conf.
Create the directory and config file
If the /etc/systemd/journald.conf file doesn't exist, create it by:
Create the journald.conf file:
sudo touch /etc/systemd/journald.conf
Configuration settings
The Systemd provides many options for you to manage the log files and by combining these parameters you can limit the disk space used by the journal files.
A list of the available options are here.
Here is a quick description of the options we're going to use for our example:
- Storage, controls where to store journal data
- SystemMaxUse, specifies the maximum disk space that can be used by the journal in persistent storage
- SystemMaxFileSize, controls how large individual journal files can grow to in persistent storage before being rotated
- RuntimeMaxUse, control how much disk space the journal may use up at most
You should open the /etc/systemd/journald.conf file in your favorite text editor and put:
[Journal]
Storage=persistent
SystemMaxUse=1G
SystemMaxFileSize=100M
RuntimeMaxUse=100M
Here, we set 1G and 100M, which means 1 Gigabyte and 100 Megabytes. You can also use K for Kbytes, amongst others.
After the changes, you have to restart the journald after updating the file. To restart use the command:
sudo systemctl restart systemd-journald
You can verify the integrity of the log files by running:
sudo journalctl --verify
Logrotate
Logrotate is a tool to manage the log files created by a system processes. It can automatically compress, rename, remove logs and more for your convenience and save your system's resources. Log files can be handled timely, or when it grows too large.
Prerequesite
The logrotate tools is available by default on Ubuntu.
We're using Ubuntu for our guide to keep it simple. If you are using a different and support operating system make sure you install Logrotate before proceeding.
You can check if logrotate is installed by executing:
logrotate --version
At the time this guide was written, we got the following output:
logrotate 3.21.0
Default mail command: /usr/bin/mail
Default compress command: /bin/gzip
Default uncompress command: /bin/gunzip
Default compress extension: .gz
Default state file path: /var/lib/logrotate/status
ACL support: yes
SELinux support: yes
If you run an earlier or older version, changes or tweaks might apply.
Explore the Configuration files
The configuration files we'll be exploring today are found in the following locations in Ubuntu:
-
/etc/logrotate.conf, is the main configuration settings file. It includes the statement to pull in configuration files from other directories, e.g. the
/etc/logrotate.d -
/etc/logrotate.d/, a directory where packages drop log rotation information
We use Ubuntu for our guide, if you are on a different distro, you have to determine the file configuration file locations
The content of the file /etc/logrotate.conf should be similar to:
# see "man logrotate" for details
# global options do not affect preceding include directives
# rotate log files weekly
weekly
# use the adm group by default, since this is the owning group
# of /var/log/.
su root adm
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
#dateext
# uncomment this if you want your log files compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# system-specific logs may also be configured here.
The configuration settings if anything like the above, tell us that the rotation happens weekly, keeping 4 weeks worth of backlogs, etc.
To learn more about other configuration options consult the logrotate manual page.
Create the Lightning logrotate configuration file
Let's create a Logrotate configuration file for Fleek Network Lightning Service. Create the file by running the command:
sudo touch /etc/logrotate.d/lightning
Lightning Configuration Settings
Open the recently created file /etc/logrotate.d/lightning in your favorite text editor.
Add the following lines to the file:
/var/log/lightning/*.log {
rotate 5
daily
size 50M
notifempty
compress
}
Remember to save the file before exiting the text editor. You can test the configuration file by running the command:
sudo logrotate /etc/logrotate.conf --debug
The configuration file above declares that for the log files in the /var/log/lightning directory, the log files are rotated 5 times daily before being removed, as long they grow bigger than 50 megabytes in file size or empty. Old versions of log files are compressed with gzip.
Bear in mind that this configuration file also inherit the default behavior, e.g. the create as set in the main configuration file /etc/logrotate.conf.
Feel free to customize the settings to your liking by checking the documentation in the logrotate manual page.
Cron job
Depending on the operating system, you have to set up a cron job to execute logrotate with the configuration file daily. Since we are using Ubuntu for our example, a daily cron job runner is already set up for us. If you are on a different Distro/OS make the required amends.
Verify that the /etc/cron.daily/logrotate exists and includes the execution of logrotate with the configuration argument /etc/logrotate.conf.
/usr/sbin/logrotate /etc/logrotate.conf
If you have modified the location of the binary or main configuration file, make sure this is set correctly to your custom locations.
To summarize, the logrotate /etc/logrotate.conf is executed and as logrotate.conf goes through its list of commands, it calls include /etc/logrotate.d. It means that any scripts in /etc/logrotate.d are executed, such as the Lightning Configuration Settings.
Conclusion
The guide starts by warning us about the stream text messages that the Fleek Network emits by default. As the text data is aggregated and stored in the file system it can lead to fill up the limited available disk space quickly, causing issues to the operation of the system.
To help control it, the journald is introduced, by explaining its role as a centralized message system, that runs alongside the application logs. Then, have it configured to limit the maximum file size, amongst other system-wide settings.
Finally, logrotate is discussed in helping us manage the Fleek Network Lightning application log files by setting it to automatically compress, rename, remove logs for the system admin convenience and saving system's resources.
